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Abstract 

In  attempting  to  develop  a  performance  measurement  system  for  assessing  the 
effectiveness  of  Integrated  Product  Teams  (IPTs),  this  research  focused  on  the 
identification  of  those  characteristics  that  were  most  important  to  the  customers  of  IPTs. 
Customer  satisfaction  was  selected  as  the  focal  point  because  it  is  the  foundation  of  the 
Integrated  Weapon  System  Management  (IWSM)  philosophy  which  fostered  the  IPT 
approach  to  conducting  business.  The  customer  satisfaction  data  was  gathered  from  the 
Quality  Air  Force  self-assessments  conducted  by  the  B-l,  B-2,  F-15,  F-16,  F-22,  and  C-17 
System  Program  Offices,  as  well  as  interviews  conducted  by  Brigadier  General  Kenne  with 
several  of  the  major  customers  of  the  Aeronautical  Systems  Center.  The  seven 
characteristics  identified,  through  the  analysis  of  the  data  gathered,  provide  the  initial  step 
in  the  development  of  a  performance  measurement  system  for  assessing  the  effectiveness 
of  IPTs. 
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USING  CUSTOMER  SATISFACTION  FOR  MEASURING  THE 


EFFECTIVENESS  OF  INTEGRATED  PRODUCT  TEAMS 


I.  Introduction 


Background 

The  merger  of  Air  Force  Systems  Command  (AFSC)  and  Air  Force  Logistics 
Command  (AFLC)  into  the  Air  Force  Materiel  Command  (AFMC)  created  a  need  to 
develop  new  ways  of  doing  business.  “As  AFSC  and  AFLC  integrated  their  activities, 
both  commands  learned  that  although  they  used  the  same  words,  they  were  not  speaking 
the  same  language.  For  example,  ‘requirements’  in  AFSC  meant  the  ‘the  user’s  needs’ 
stated  so  that  AFSC  knew  what  they  were  expected  to  deliver  to  the  customer.  In  AFLC, 
‘requirements’  meant  ‘how  many  parts  and  supplies’  were  needed  to  sustain  the  user 
during  peace  and  war”  (AFMC,  1993: 9).  Air  Force  leadership  realized  the  need  to 
implement  a  new  management  philosophy  in  order  to  meet  the  new  challenges  presented 
by  the  merger.  The  management  philosophy  developed  was  Integrated  Weapon  System 
Management  (IWSM).  “Integrated  Weapon  System  Management  is  the  AFMC 
management  philosophy  for  acquiring,  evolving,  and  sustaining  our  products.  It 
empowers  a  single  manager  with  authority  over  the  widest  range  of  decisions  and 
resources  to  satisfy  customer  requirements  throughout  the  life  cycle  of  the  product” 
(AFMC,  1993:  9).  IWSM  is  centered  around  eight  interacting  elements.  One  of  the 
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primary  elements  of  IWSM,  and  certainly  the  most  relevant  to  this  research,  is  Integrated 
Product  Development  (IPD). 

IPD  “systematically  employs  a  teaming  of  functional  disciplines  to  integrate  and 
concurrently  apply  all  necessary  processes  to  produce  an  effective  and  efficient  product 
that  satisfies  customer’s  needs”  (AFMC,  1993:  275).  In  addition,  IPD  “requires  a  product 
focus  and  a  complete  understanding  of  the  processes  required  to  optimize  the  product. 

The  products  delivered  will  dictate  the  ultimate  success  of  our  efforts  in  the  eyes  of  the 
customer”  (AFMC,  1993:  276).  In  a  White  Paper,  dated  21  April  1993,  General  Ronald 
W.  Yates,  AFMC  Commander,  wrote  “The  IPD  philosophy  and  its  basic  tools  will  enable 
this  command  to  realize  the  full  benefit  of  Integrated  Weapon  System  Management 
(IWSM)”  (AFMC,  1993:  369). 

The  teaming  concept  emphasized  in  the  IPD  philosophy  was  implemented  through 
the  formation  of  Integrated  Product  Teams  (IPTs).  EPTs  are  “cross-functional  teams  that 
are  formed  for  the  specific  purpose  of  delivering  a  product  or  managing  a  process  for  the 
customer.  The  team  is  not  the  end  goal  of  IPD,  but  rather  the  means  through  which  much 
of  the  work  in  an  organization  gets  done”  (AFMC,  1993:  288). 

As  commander  of  the  Advanced  Tactical  Fighter  System  Program  Office  in  1990, 
Lieutenant  General  James  A.  Fain,  acting  under  the  direction  of  General  Yates,  became 
the  pioneer  for  the  implementation  of  IPTs  as  the  primary  working  unit  in  a  major  weapon 
system  acquisition  in  the  Aeronautical  Systems  Center  (ASC).  In  1994,  as  Commander  of 
ASC,  Lieutenant  General  Fain  directed  that  IPTs  would  be  used  for  all  procurement 
initiatives  in  ASC.  This  direction  required  all  program  offices  to  abandon  the  functional 
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organizational  structure  previously  used  and  adopt  an  organizational  structure  that  was 
more  appropriately  suited  for  IPT  implementation.  Each  IPT  consisted  of  a  team  leader, 
who  was  held  responsible  for  the  ultimate  success  or  failure  of  the  program,  and  a 
dedicated  cadre  of  functional  specialists  whose  role  was  to  support  the  team  leader  in  the 
manner  required  to  accomplish  the  program  objectives. 

The  foundation  for  all  three  tiers  of  the  new  management  philosophy  (IWSM,  IPD, 
and  IPT)  is  customer  satisfaction.  The  IWSM  concept  emphasizes  the  need  to  empower  a 
single  manager  with  the  authority  to  satisfy  customer  needs.  The  IPD  concept  focuses  on 
the  teaming  of  functional  disciplines  to  produce  an  effective  and  efficient  product  that 
satisfies  customer  needs.  And  finally,  the  IPTs  are  the  cross- functional  teams  with  the 
specific  purpose  of  delivering  a  quality  product  for  the  customer.  Because  of  the 
enormous  emphasis  at  all  levels,  customer  satisfaction  would  appear  to  be  the  best 
foundation  for  the  development  of  metrics  for  measuring  IPT  effectiveness. 

Statement  of  the  Problem 

Since  IPTs  are  a  relatively  new  concept  to  the  ASC  environment,  it  is  imperative 
that  a  set  of  performance  measurements  be  established  that  will  accurately  indicate  the 
quality  of  IPT  performance.  These  performance  measures,  or  metrics,  must  focus  on 
those  characteristics  which  are  determined  to  be  most  critical  to  customer  satisfaction. 
Equally  important,  the  characteristics  found  to  be  most  critical  to  customer  satisfaction 
must  be  consistent  with  the  criteria  of  good  metrics  as  defined  in  the  AFMC  Metric 
Handbook. 
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A  recent  research  effort  (Casipit  and  Hodgkiss,  1994)  studied  the  customer 
satisfaction  programs  of  six  quality  award  winning  service  organizations  in  the  military  and 
commercial  sectors.  ASC  was  one  of  the  six  organizations.  The  Casipit  and  Hodgkiss 
study  centered  around  a  series  of  questionnaires  pertaining  to  developing,  tracking,  and 
using  customer  satisfaction  inputs  and  outputs  within  each  of  the  organizations.  As  an 
organization  made  up  of  a  conglomeration  of  program  offices,  the  responses  provided  by 
ASC  were  based  on  the  input  of  those  individual  program  offices  rather  than  on  a  front 


office  assessment  of  the  appropriate  responses.  In  light  of  that  approach,  and  since  IPTs 
are  the  primary  working  unit  of  the  program  offices,  the  responses  provided  by  ASC  can 
appropriately  be  extrapolated  to  the  individual  IPTs  within  each  program  office.  The 
ASC  responses  that  are  driving  this  research  are: 


Question  A3:  Do  you  have  corporate-wide  buy-in  for  your  customer  satisfaction 
program? 

ASC:  Bottom  line  answer  is  no-  it  is  not  corporate  wide  (Casipit  and  Hodgkiss, 

1994:  C-4). 


Question  B 1:  Did  you  develop  your  own  customer  satisfaction  measure  and 
methods  to  suit  your  organization’s  needs  or  did  you  essentially 
copy  another  organization’s? 

ASC:  Some  of  both.  Some  of  the  SPO’s  and  other  2  letter  offices  developed 

their  own  measures  and  methods  to  suit  the  needs  of  their  specific 
customers  and  situation.  Others  simply  copied  generic  measures  and 
methods  (surveys)  from  others  (basically  to  be  in  compliance  with  ASC 
requirement  that  all  organizations  must  measure  customer  satisfaction. 
Again,  for  ASC,  the  answer  is  “it  depends”  (Casipit  and  Hodgkiss,  1994: 
C-ll). 


Question  B5:  Do  you  determine  the  validity  of  these  measures? 
ASC:  No,  not  really  (Casipit  and  Hodgkiss,  1994:  C-15). 
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Question  C2:  Do  your  customers  help  develop  these  measures?  If  so,  how. 

ASC:  Yes  in  some  cases  and  no  in  others.  The  B-2  CARD  system  and  the 

LANTIRN  SPO  both  developed  their  instruments  (measures  and  methods) 
with  the  input  from  their  customers.  Other  SPOs  or  offices  developed 
measures  without  input  from  customers  (Casipit  and  Hodgkiss,  1994:  C- 
23). 

A  review  of  the  ASC  responses  indicates  that  the  development  of  customer 
satisfaction  programs  and  measurement  approaches  within  the  individual  program  offices 
is  currently  at  widely  varying  levels. 

While  AFMC  organizations  currently  have  standard  metrics  to  identify,  track,  and 
report,  in  the  areas  of  cost,  schedule,  and  performance,  at  the  program  level,  there  are  no 
metrics  in  place  to  measure  the  effectiveness  of  IPTs  within  the  program  office.  The 
development  and  implementation  of  meaningful,  customer-oriented  metrics  would  allow 
management  to  identify  the  current  level  of  EPT  performance.  For  this  to  occur,  the 
attributes  necessary  to  achieve  customer  satisfaction,  on  the  IPT  level,  must  be  identified, 
developed,  and  validated. 

Scope 

The  purpose  of  this  research  is  to  facilitate  the  development  of  metrics  for 
assessing  the  effectiveness  of  IPTs.  Customer  satisfaction  is  the  key  to  the  management 
philosophy  implemented  by  IWSM  and,  as  such,  must  be  the  central  focus  in  the 
development  of  such  metrics.  This  research  will  attempt  to  transform  the  needs  of  IPT 
customers  into  a  consolidated  group  of  important  attributes  necessary  to  determine  the 
effectiveness  of  the  IPT  as  a  working  unit.  To  accomplish  this  task,  the  following 
questions  will  guide  this  research. 
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Research  Questions 


1.  Why  measure  customer  satisfaction? 

2.  What  does  an  LPT  in  ASC  consist  of,  and  how  do  they  operate? 

3.  What  are  the  key  characteristics  of  effective  teams? 

4.  Who  are  the  customers  of  ASC’s  IPTs? 

5.  What  characteristics  of  performance  do  IPTs  and  IPT  customers  perceive  as 
most  important? 

6.  Of  those  characteristics  identified  by  the  IPTs  and  their  customers  as  most 
important,  which  are  the  most  appropriate  as  a  foundation  for  the  development  of 
metrics  for  measuring  the  effectiveness  of  IPTs? 

These  characteristics  will  provide  a  foundation  for  the  subsequent  development  of  specific 
customer  satisfaction  metrics  and  will  enable  future  researchers  to  more  clearly  focus  their 
efforts  on  these  areas. 

The  first  three  research  questions  will  be  addressed  through  a  review  of  the  current 
literature,  in  Chapter  II,  as  it  pertains  to  each  individual  subject.  The  final  three  research 
questions  will  be  addressed  through  the  analysis  of  information  collected  as  part  of  the 
data  collection  process  described  in  Chapter  HI. 
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II.  Literature  Review 


Introduction 

As  noted  in  Chapter  I,  the  IWSM  philosophy  has  placed  enormous  emphasis  on 
customer  satisfaction  as  a  quality  measure.  Customer  satisfaction  is  today’s  biggest  issue 
in  the  quality  improvement  movement  and  in  business  as  a  whole  (Casipit  and  Hodgkiss, 
1994:  2-2).  This  literature  review  will  address  the  importance  of  measuring  customer 
satisfaction  and  its  effect  on  the  business  world  as  well  as  the  Department  of  Defense. 

Also,  in  order  to  measure  customer  satisfaction,  a  complete  understanding  of  the 
subject  to  be  measured  (IPTs),  as  well  as  a  complete  understanding  of  the  measuring 
instrument  (metrics)  is  essential.  This  literature  review  describes  the  philosophy  of  IPTs 
and  their  role  in  ASC  organizations.  In  turn,  the  need  for  effective  team  building  and 
effective  teams  are  explored  in  great  detail  in  order  to  establish  a  foundation  for  the 
development  of  the  most  appropriate  effectiveness  metrics  for  IPTs.  Finally,  a  summary 
of  the  literature  review  is  presented. 

Why  Measure  Customer  Satisfaction? 

Customer  satisfaction  is  the  cornerstone  of  the  quality  improvement  movement. 
Armand  Feigenbaum,  a  noted  authority  on  the  subject,  defines  quality  as  “the  total 
composite  product  and  service  characteristics  of  marketing,  engineering,  manufacturing 
and  maintenance  through  which  the  product  or  service  in  use  will  meet  the  expectations  of 
the  customer ”  (Feigenbaum,  1983: 7).  This  definition  raises  several  important 
implications: 
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1.  Satisfaction,  hence  quality,  depends  upon  evaluations  made  by  persons  external 
to  the  organization  that  produces  the  goods  or  services.  It  is  not  determined  by 
some  internal  standard  -  unless  that  standard  has  been  carefully  established  on  the 
basis  of  accurate  interpretations  of  customer  information  and  feedback  about  their 
needs  and  expectations. 

2.  Customer  satisfaction  usually  relates  to  long-term  use  of  goods  or  results  of  a 
service,  so  its  evaluation  is  based  on  comparison,  over  considerable  time,  with  the 
customer’s  perception  of  competitive  offerings. 

3.  Quality  requires  a  composite  of  attributes  to  satisfy  a  range  of  expectations  of 
numerous  customers  or  potential  customers.  Quality  often  means  different  things 
to  different  people.  (Dilworth,  1993:  463) 

In  a  manufacturing/product  environment,  a  person  evaluates  at  least  eight  aspects 
of  a  product  of  service  to  see  if  it  satisfies  his  or  her  needs.  These  aspects  are: 

1.  Performance  -  such  as  color  or  clarity  of  a  television. 

2.  Features  -  such  as  stereo  sound  or  remote  control. 

3.  Reliability  -  how  often  the  product  will  need  repair. 

4.  Serviceability  -  ease,  expense,  and  time  required  to  repair. 

5.  Durability  -  length  of  product  life. 

6.  Conformance  -  how  well  the  product  meets  the  specifications  set  by  the 
designers. 

7.  Aesthetic  characteristics  -  how  the  item  looks,  smells,  feels,  etc. 


8 


8.  Perceived  quality  -  feeling  of  confidence  based  on  company  reputation  or 
customers  prior  use  of  the  product.  (Garvin,  1987:  101) 

Service  quality,  however,  is  not  as  easy  to  describe  in  quantifiable  measures  that 
can  be  used  within  a  company  to  see  if  employees  are  working  in  a  consistent  and  correct 
manner.  Five  major  attributes  found  to  be  important  in  describing  service  quality  are: 

1.  Reliability  -  the  ability  to  perform  the  promised  service  dependably  and 
accurately. 

2.  Responsiveness  -  the  willingness  to  help  customers  and  provide  prompt  service. 

3.  Tangibles  -  physical  facilities,  equipment,  and  the  appearance  of  personnel. 

4.  Assurance  -  the  knowledge  and  courtesy  of  employees  and  their  ability  to 
convey  trust  and  confidence. 

5.  Empathy  -  the  caring,  individual  attention  provided  to  customers.  (Zeithamal, 
Parasuraman,  and  Berry,  1990:26) 

A  service  organization  must  identify  specific  attributes  that  are  important  and  the  degree 
to  which  they  are  important  in  satisfying  its  customers  (Dilworth,  1993:465).  For 
example.  Metropolitan  Life  Insurance  Company  used  surveys  to  determine  customer 
expectations  and  their  perceptions  of  the  performance  that  was  provided  for  the  five 
major  attributes  of  service  quality  listed  above.  By  identifying  what  is  important  to  its 
customer,  the  company  can  establish  performance  standards  so  its  employees  know  what 
they  are  to  do  (Falzon,  1988:  55).  “It  may  appear  that  quality  is  very  expensive  to  achieve 
with  so  many  attributes  of  the  product  to  consider  and  all  parts  of  a  company  involved  in 
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trying  to  satisfy  many  customers.  Quality  does  cost-but  the  failure  to  achieve  quality  may 
cost  even  more”  (Dilworth,  1993:465). 

Improving  customer  satisfaction  is  one  of  the  most  important  challenges  facing 

businesses  in  the  1990s  (Barsky  and  Labaugh,  1992:  32).  As  Peterson  and  Wilson  point 

out,  it  is  the  primary  goal  of  any  business.  According  to  Peterson  and  Wilson: 

For  a  business  to  be  successful  in  the  long  run,  it  must  satisfy  customers,  albeit  at  a 
profit.  Indeed,  it  can  be  argued  that  satisfying  customers  is  the  primary  obligation 
of  a  company.  Hence,  customer  satisfaction  is  a  defensible  and  appropriate 
company  objective  -  the  glue  that  holds  various  corporate  functions  together  and 
directs  corporate  resource  allocation.  Conceptually,  virtually  all  company 
activities,  programs,  and  policies  should  be  evaluated  in  terms  of  their  contribution 
to  satisfying  customers.  (Peterson  and  Wilson,  1992:  61) 

The  Department  of  Defense  also  emphasizes  the  fact  that  the  customer  defines 

quality  and  that  the  organization  exists  to  satisfy  the  customer.  According  to  the  DOD 

TQM  implementation  guide: 

The  customer  defines  the  purpose  of  the  organization  and  every  process  within  it. 
Success  means  striving  to  become  the  best  supplier  of  your  particular  products  and 
services  in  the  minds  of  those  customers.  To  achieve  that  success,  your 
organization  must  align  its  overriding  strategic  vision  with  a  vision  of  customer 
service  and  satisfaction.  ( DOD,  1990:  2-12) 

With  such  evidence,  it  is  obvious  why  organizations  must  explicitly  and  actively  address 

customer  satisfaction  in  the  course  of  daily  business. 

The  reasons  for  measuring  customer  satisfaction  can  be  categorized  along  three 

lines:  customer  behavior,  increased  competition,  and  Governmental  reform  (Casipit  and 

Hodgkiss,  1994:  2-3). 

Customer  behavior  provides  a  strong  impetus  to  measure  customer  satisfaction. 
Failure  to  ascertain  the  satisfaction  level  of  your  customers  may  result  in  the  loss  of  their 
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business.  Today’s  customer  expect  higher  quality  in  products  and  services  and  have  a 
much  lower  tolerance  for  poor  service  and  goods. 

Similarly,  the  increased  level  of  competition  that  organizations  are  facing  today, 
particularly  from  the  international  market  place,  makes  it  imperative  that  they  be 
constantly  in  touch  with  the  satisfaction  level  of  their  customers.  Government 
organizations,  such  as  aviation  depots,  are  facing  stiff  competition  with  contractors  and 
other  government  depots  in  the  face  of  budget  cutbacks  (Federal  Quality  Institute,  1993: 
2-1). 

And  finally.  Governmental  reform  is  mandating  the  measurement  of  customer 
satisfaction  into  the  process.  Executive  Order  12862  -  Setting  Customer  Service 
Standards  was  signed  by  President  Clinton  on  1 1  September  1993.  The  order  requires 
Government  agencies  to  have  customer  focus  and  reward  performance  and  allocate 
resources  based  on  customer  satisfaction  results  (Clinton,  1993:  1737-1738).  In  addition, 
when  Congress  passed  the  Government  Performance  and  Results  Act  of  1993,  all 
Government  agencies  were  required  to  increase  the  level  of  emphasis  given  to  service 
quality  and  customer  satisfaction.  More  specifically,  the  act  requires  that  “effectiveness 
measures  should  be  direct  indicators  of  the  final  product  or  service  as  observed  by  the 
customer”  (United  States  Senate,  1993: 15). 

It  should  be  evident  from  the  preceding  discussion  that  measuring  customer 
satisfaction  is  done  to  improve  processes,  better  satisfy  the  customer,  and  add  value  to  the 
product  or  service  (Casipit  and  Hodgkiss,  1994:  2-8). 


11 


IPT  Philosophy  and  ASC  Organizations 


The  United  States  Air  Force  defines  Integrated  Product  Development  (IPD)  as: 

“A  philosophy  that  systematically  employs  a  teaming  of  functional  disciplines  to  integrate 
and  concurrently  apply  all  necessary  processes  to  produce  an  effective  and  efficient 
product  that  satisfies  customer  needs”  (AFSC,  1991:  2-1).  In  the  case  of  system  program 
offices  at  ASC,  this  teaming  approach  is  implemented  through  the  use  of  IPTs  to  acquire 
all  major  weapon  systems.  The  IPTs  consist  of  a  team  leader,  who  is  responsible  for  the 
ultimate  success  or  failure  of  the  program,  and  a  dedicated  cadre  of  functional  specialists 
whose  role  is  to  support  the  team  leader  in  the  manner  required  to  accomplish  the  program 
objectives. 

ASC  organizations  operate  on  a  project  basis.  Each  new  project  requires  the 
formation  of  a  project  group  (IPT)  consisting  of  a  team  leader  and  members  from  all 
appropriate  functional  areas.  “The  project  manager  creates  a  project  group  by  negotiating 
with  functional  managers  and  then  ‘borrowing’  the  expertise  and  physical  resources 
needed  for  the  project”  (Nicholas,  1990:  152).  The  project  manager,  or  IPT  team  leader, 
integrates  the  support  efforts  of  each  team  member  to  meet  project  goals.  As  projects  are 
completed,  all  team  members  return  to  their  functional  areas  for  reassignment  to  new 
projects.  This  approach  provides  many  benefits  to  the  overall  organization.  Some  of  the 
specific  benefits  are: 

1.  Knowledge  transfer.  Workers  share  project  experiences  with  colleagues 
in  their  fields  of  specialization  which  keeps  them  current  in  their 
profession. 
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2.  Team  formation.  IPT  membership  can  be  tailored  rather  easily  to 
accommodate  unique  projects  or  goals. 

3.  Schedules  and  resources.  In  a  multiproject  organization,  schedules  and 
resource  requirements  can  be  maintained  and  tracked  for  several 
projects  at  once.  (Nicholas,  1990:  153) 

Characteristics  of  Effective  Teams 

AFMC  Pamphlet  800-60,  Integrated  Weapon  Systems  Management  Guide, 
identifies  team  building  as  key  to  the  success  of  any  project.  “You  must  establish  a  solid, 
dedicated,  and  knowledgeable  team,  consisting  of  key  members  from  the  acquisition 
program  office  and  support  program  office”  (AFMC,  1993:  21).  Effective  team  building 
does  not  ensure  success,  but  without  it  success  cannot  be  attained.  This  concept  is  central 
to  the  thinking  behind  the  IPT  approach;  a  dedicated  team  of  functional  specialists, 
working  in  close  coordination  and  under  the  direction  of  a  single  leader. 

Different  researchers  have  measured  team  effectiveness  in  a  variety  of  ways.  Two 
commonly  found  categories  of  team  effectiveness  are: 

1.  Group  interdependence  -  how  closely  team  members  work  together. 

2.  Outcome  interdependence  -  commitment  of  team  members  to 
achievement  of  project  goals  and  customer  satisfaction. 

Group  Interdependence.  Group  interdependence  is  measured  as  the  degree  of 
task-driven  interaction  among  group  members  (Guzzo  and  Shea,  1987: 138).  Beyond 
actual  physical  time  spent  together,  group  interdependence  encompasses: 
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1.  Effective  communication  among  members. 

2.  Effective  ways  of  resolving  group  process  problems. 

3.  Techniques  for  using  conflict  in  a  constructive  way. 

4.  Greater  collaboration  and  creativity  among  team  members. 

5.  A  more  trusting,  supportive  atmosphere  within  the  group. 

6.  Clarification  of  the  team’s  purpose  and  the  role  of  each  team  member. 

(Nicholas,  1990:  205) 

Outcome  Interdependence.  Successful  projects  are  the  result  of  teams  which 
succeed  in  achieving  what  they  set  out  to  do.  The  prominent  feature  found  in  all 
successful  teams  is  that  they  know  what  must  be  done  and  are  committed  to  doing  it.  A 
study  conducted  by  Peter  Vaill  identified  three  characteristics  always  present  in  the 
behavior  and  attitudes  of  members  of  high  performing  teams;  time,  feeling,  and  focus 
(Nicholas,  1990:  202-203).  First,  members  devote  extraordinary  amounts  of  time  to  the 
task.  Second,  they  have  very  strong  feelings  about  the  attainment  of  the  goal.  And  third, 
they  focus  on  key  issues  and  have  a  clear  list  of  priorities  in  mind.  In  addition,  Vaill  found 
that: 

1.  Commitment  to  the  purpose  of  the  system  is  never  perfunctory  and 

motivation  is  always  high. 

2.  Teamwork  is  focused  on  the  task.  Members  develop  behaviors  that 

enable  them  to  what  they  must  to  accomplish  the  task. 

3.  Leadership  is  strong,  clear,  and  never  ambivalent. 

4.  The  system  is  clearly  bounded  from  other  systems;  members  have  a 

consciousness  that  they  are  “different”. 
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In  addition  to  accomplishment  of  the  task,  effective  teams  strive  to  maintain  the 
integrity  of  the  group;  each  aspect  is  dependent  on  the  other.  The  failure  of  one  team 


member  to  fully  commit  to  the  task  may  result  in  violations  of  aspects  such  as 
communication  and  trust,  with  a  resulting  failure  to  accomplish  the  primary  task. 

Criteria  of  Good  Metrics 

Why  metrics?  Bill  Hewlett,  a  founder  of  Hewlett-Packard,  used  to  say  ‘“You 
cannot  manage  what  you  cannot  measure’,  and  his  corollary  was  ‘What  gets  measured 
gets  done’”  (House  and  Price,  1991:  93).  The  challenge  of  any  organization  is  to  develop 
performance  measures  that  will  help  the  organization  assess  how  well  or  how  poorly  they 
are  doing.  These  measures  must  also  motivate  action  to  continually  improve  all  business 
practices.  Metrics  are  nothing  more  than  meaningful  measures.  The  AFMC  Metrics 
Handbook  defines  metrics  as  “a  measurement  made  over  time,  which  communicates  vital 
information  about  the  quality  of  a  process,  activity,  or  resource”  (AFMC,  1993:  3).  The 
handbook  identifies  eight  criteria  of  a  good  metric: 

1.  It  is  accepted  as  meaningful  to  the  customer. 

2.  It  tells  how  well  organizational  goals  and  objectives  are  being  met 

through  processes  and  tasks. 

3.  It  is  simple,  understandable,  logical,  and  repeatable. 

4.  It  shows  a  trend. 

5.  It  is  unambiguously  defined. 

6.  Its  data  is  economical  to  collect. 

7.  It  is  timely. 
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BUT  MOST  IMPORTANTLY: 


8.  It  drives  the  appropriate  action. 

Meaningful  to  the  Customer.  “Organizational  literature  usually  has  defined 
quality  to  mean  meeting  a  preset  standard  or  specification.  TQM,  however,  defines 
quality  as  whatever  the  customer  says  is  quality”  (Kinlaw,  1992:  29).  Kinlaw’s  statement 
on  quality  certainly  holds  true  for  the  development  of  good  metrics.  The  customer  is  the 
ultimate  user  of  the  information  and,  therefore,  the  final  judge  as  to  its  usefulness.  Metrics 
which  are  designed  to  provide  information  which  is  of  value  only  to  the  organization,  and 
not  the  customer,  are  of  no  value  to  the  quality  improvement  process. 

Meeting  Organizational  Goals  and  Objectives.  Metrics  are  not  goals  or 
objectives.  Metrics  measure  various  levels  of  goals  and  objectives,  but  they  do  not,  in  and 
of  themselves,  contribute  to  the  actual  achievement  of  the  goals  and  objectives  of  the 
organization.  Metrics  do  foster  the  achievement  of  goals  and  objectives  through  process 
improvement.  Metrics  must  focus  the  attention  of  management  on  the  areas  which  will 
result  in  the  achievement  of  the  goals  and  objectives  of  the  organization  (Hamner  and  La 
Fleur,  1993:  2-3). 

Simple,  Understandable,  Logical,  and  Repeatable.  “Measurement  will  only 
take  root  in  our  profession  if  it  is  relatively  easy  to  do  and  provides  valuable  insights” 

(Fisk  and  Ford,  1993:  40).  Metrics  must  be  simple  enough  to  be  understood  by  everyone 
involved  in  the  process,  most  importantly  the  customer.  The  logic  inherent  in  the  metric 
must  be  clear  and  well  defined.  In  order  to  be  repeatable,  a  metric  must  provide  the  same 
conclusion  at  a  given  point  in  the  process  regardless  of  the  individual  performing  the 
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measurement  (Hamner  and  La  Fleur,  1993:  2-4).  Repeatability  is  essential  if  the 
measurement  is  to  be  meaningful  to  the  customer. 

It  Shows  a  Trend.  One  of  the  main  purposes  of  a  good  metric  is  to  facilitate 
continuous  improvement.  Accordingly,  a  good  metric  must  reflect  progress,  or  lack 
thereof,  toward  improved  quality  by  indicating  if  quality  is  converging  or  diverging  toward 
a  goal  (Juran,  1988:  77).  Measurement  over  time  is  essential  to  determining  the  overall 
health  of  a  process.  The  trends  that  can  be  detected  via  the  measurement  process  are 
essential  tools  that  allow  management  to  concentrate  their  efforts  on  the  areas  which 
require  their  attention.  A  metric  must  provide  management  with  a  means  of  distinguishing 
the  true  problem  areas. 

It  is  Unambiguously  Defined.  It  is  imperative  that  a  metric  be  unambiguously 
defined  so  that  measurements  at  various  points  in  time  can  be  compared  against  the  same 
baseline.  A  metric  must  define: 

1.  Who  will  take  and  record  the  measurement. 

2.  What  exactly  is  to  measured. 

3.  When,  or  at  what  point  in  the  process,  the  measurement  is  to  be  taken. 

Any  vagueness  or  complexity  becomes  a  natural  source  of  divisiveness. 

(Juran,  1988:  76) 

A  metric  which  is  defined  so  ambiguously  that  successive  measurements  cannot  be 
used  for  comparison  purposes  is  of  no  value  to  the  quality  improvement  process. 

Its  Data  is  Economical  to  Collect.  The  metric  must  add  value  to  the  process. 
“The  unit  of  measure  should  be  established  at  that  level  of  precision  which  enables  us  to 
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make  valid  decisions  from  the  data.  To  go  beyond  that  level  of  precision  usually  adds  cost 
without  adding  value”  (Juran,  1988:  78).  If  the  effort  and/or  cost  associated  with 
collection  of  the  data  exceeds  the  benefit  derived  from  the  resulting  process 
improvements,  the  metric  is  of  no  value  to  the  quality  improvement  process. 

It  is  Timely.  Data  must  be  collected  and  analyzed  in  a  manner  that  allows 
management  sufficient  time  to  digest  the  data  and  influence  the  process.  The  more  critical 
the  process,  the  more  essential  it  is  that  the  data  be  timely.  A  metric  must  provide 
management  with  essential  data  in  time  to  influence  the  process. 

It  Drives  the  Appropriate  Action.  Above  all  else,  a  metric  must  drive  behavior 
that  is  attainable  and  results  in  continuous  improvement  to  the  process.  A  good  metric 
focuses  management’s  attention  on  the  true  problem  areas  and  results  in  positive  actions 
which  increase  the  organization’s  chances  of  accomplishing  its  goals.  Metrics  which  do 
not  encompass  both  of  these  attributes,  are  counterproductive  and  of  no  value  to  the 
quality  improvement  process. 

Summary 

The  initial  research  effort  focused  on  the  exploration  of  customer  satisfaction, 
IPTs,  team  building,  characteristics  of  effective  teams,  and  criteria  of  good  metrics.  The 
philosophy  of  IPTs  and  its  underlying  focus,  customer  satisfaction,  was  defined  within  the 
IWSM  philosophy.  The  research  focus  then  shifted  to  the  investigation  of  IPTs  as  the 
primary  working  unit  of  ASC.  Since  the  objective  of  this  research  is  to  facilitate  the 
development  of  metrics  to  measure  the  effectiveness  of  IPTs,  a  thorough  analysis  of 


18 


relevant  team  building,  effective  team  characteristics,  and  relevant  criteria  for  the 
development  of  metrics  was  accomplished.  These  efforts  will  provide  a  strong  basis  for 
identifying  the  attributes  that  are  most  critical  to  the  success  or  failure  of  programs 
managed  by  IPTs  and  the  subsequent  development  of  performance  measures  for  IPTs. 

Chapter  in  will  define  the  methodology  that  will  be  used  to  collect  the  data 
required  for  identification  of  the  customer  satisfaction  attributes  that  are  most  critical  to 
the  success  or  failure  of  EPT  programs. 
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III.  Methodolo2v 


Overview 

In  order  to  identify  those  characteristics  that  are  most  critical  to  customer 
satisfaction,  this  research  will:  (1)  identify  the  BPT  customers  and  the  criteria  currently 
used  by  individual  program  offices  to  determine  customer  satisfaction  levels;  (2)  gather 
data  on  the  characteristics  that  are  most  important  to  the  customers  of  IPTs;  and  (3) 
subject  the  data  to  analysis  and  use  the  results  of  that  analysis  to  identify  a  manageable 
number  of  characteristics  which  are  most  appropriate  for  use  in  the  development  of  a 
performance  measurement  system  for  measuring  the  effectiveness  of  IPTs. 

In  order  to  obtain  an  appropriate  cross-section  of  organizations  within  ASC,  this 
research  will  use  the  following  SPOs  as  a  representative  sample:  B-l,  B-2,  F-15,  F-16,  F- 
22,  and  C-17.  These  SPOs  were  identified  by  the  ASC  Quality  Improvement  Office 
(ASC/QI)  as  the  leaders  in  the  development  of  a  system  to  appropriately  determine  the 
level  of  customer  satisfaction  that  exists  within  each  respective  office. 

Customer  and  Criteria  Identification 

In  1994,  organizations  within  ASC  were  required  to  perform  an  internal  Quality 
Air  Force  Self-Assessment.  This  assessment  was  intended  to  provide  each  organization  an 
opportunity  to  examine  the  entire  organization  using  the  QAF  criteria  as  a  highly  objective 
assessment  tool. 

In  Section  7,  Customer  Focus  and  Satisfaction,  each  organization  was  required  to 
identify  its  primary  program  customers.  The  actual  breakdown  of  all  customers  for  each 
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individual  program  will  be  discussed  in  Chapter  IV.  In  addition,  Section  7  discusses  the 
systems/methods  created  by  each  SPO  to  monitor  customer  satisfaction  levels.  This 
information  will  provide  a  baseline  of  criteria  currently  used  to  determine  customer 
satisfaction  levels  within  ASC  organizations.  This  information  will  also  be  addressed  in 
Chapter  IV. 

Data  Collection  Plan 

The  second  stage  of  the  process  will  be  to  gather  inputs  from  those  customers 
identified  in  the  unit  self-assessments  regarding  the  characteristics  of  IPT  performance 
they  (the  customers)  consider  important.  There  are  currently  two  sources  of  this  data. 

1.  Quality  Air  Force  Self-Assessment,  Chapter  7  -  Customer  Focus  and 
Satisfaction. 

More  specifically.  Chapter  7  includes  sections  on  customer  expectations,  customer 
relationship  management,  commitment  to  customers,  customer  satisfaction  determination, 
customer  satisfaction  results,  and  customer  satisfaction  comparison. 

2.  Brigadier  General  Leslie  Kenne’s  (ASC/CV)  personal  interviews  with  five 
major  customers  of  ASC. 

The  customers  are  Air  Mobility  Command,  Air  Combat  Command,  Air  Education  and 
Training  Command,  Air  Force  Special  Operations  Command,  and  Headquarters  Air  Force 
Reserves.  Brigadier  General  Kenne  conducted  these  interviews  in  early  1995  by 
telephone.  In  some  instances.  Brigadier  General  Kenne  interviewed  more  than  one  agency 
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within  some  of  the  commands.  For  example,  AMC/LG  and  AMC/XP  were  both 
interviewed. 

Analysis  Method 

The  third  stage  of  the  process  will  be  to  analyze  the  customer  inputs  gathered  from 
the  self-assessments  and  the  Kenne  interviews  in  order  to  identify  those  characteristics 
most  critical  to  customer  satisfaction.  By  merging  the  inputs  from  both  sources,  a 
common  ground  between  the  existing  criteria  and  the  actual  customer  needs  can  be 
established.  In  turn,  this  commonality  will  isolate  those  characteristics  that  are  identified 
as  being  most  critical  to  customer  satisfaction  from  the  standpoint  of  both  the  IPTs  and 
the  customers.  The  characteristics  which  can  be  isolated  will  then  be  identified  as  the  key 
characteristics  against  which  metrics  should  be  developed  for  the  purpose  of  measuring 
the  effectiveness  of  IPTs. 

Chapter  IV  begins  the  actual  synthesis  of  customer  satisfaction  data  in  ASC.  It  is 
focused  on  the  identification  of  specific  customers,  retrieval  of  appropriate  customer 
satisfaction  data,  and  analysis  of  the  data.  The  format  of  Chapter  IV  will  involve 
answering  the  research  guiding  questions  that  were  identified  in  Chapter  I. 
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IV.  Findings 


Overview 

As  noted  in  Chapter  1,  research  questions  1, 2,  and  3  were  addressed  in  Chapter 
II,  literature  Review.  A  brief  synopsis  of  the  findings  for  these  three  questions  is 
provided  in  this  chapter  in  order  to  maintain  consistency  of  the  research  effort.  Research 
questions  4, 5,  and  6  will  be  addressed  following  the  synopses  of  the  previous  three 
findings. 

Research  Questions 

Research  Question  1.  Why  measure  customer  satisfaction?  Customer 
satisfaction  is  the  cornerstone  of  the  quality  improvement  movement.  We  must  measure 
customer  satisfaction  in  order  to  monitor  our  progress  in  improving  the  quality  of  our 
product  or  service.  Ultimately,  we  measure  customer  satisfaction  in  order  to  improve 
processes,  better  satisfy  the  customer,  and  add  value  to  the  product  or  service. 

Research  Question  2.  What  does  an  IPT  in  ASC  consist  of,  and  how  do  they 
operate?  An  IPT  in  ASC  consists  of  a  team  leader,  who  is  responsible  for  the  ultimate 
success  or  failure  of  the  program,  and  a  dedicated  cadre  of  functional  specialists  whose 
role  is  to  support  the  team  leader  in  the  manner  required  to  accomplish  the  program 
objectives. 

Research  Question  3.  What  are  the  key  characteristics  of  effective  teams? 

Two  commonly  found  categories  of  team  effectiveness  are: 

1.  Group  interdependence  -  how  closely  team  members  work  together. 
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2.  Outcome  interdependence  -  commitment  of  team  members  to 
achievement  of  project  goals  and  customer  satisfaction. 

Group  interdependence  is  measured  as  the  degree  of  task-driven  interaction  among 
group  members.  Beyond  the  physical  time  spent  together,  group  interdependence  is 
concentrated  on  how  well  team  members  work  with  one  another.  Outcome 
interdependence  relates  more  to  the  dedication  of  the  team  members  in  achieving  what 
they  set  out  to  do.  In  addition  to  accomplishment  of  the  task,  effective  teams  strive  to 
maintain  the  integrity  of  the  group.  The  failure  of  one  team  member  to  fully  commit  to  the 
task  may  result  in  violations  of  aspects  such  as  communication  and  trust,  with  a  resulting 
failure  to  accomplish  the  primary  task. 

Research  Question  4.  Who  are  the  customers  of  ASC’s  IPTs?  In  Section  7  of 
the  QAF  Self-Assessment,  each  organization  was  required  to  identify  its  primary  program 
customers.  Table  1  is  a  compilation  of  the  responses. 


TABLE  1 

PRIMARY  CUSTOMERS 


Program  Customers 

B-l  • 

Air  Combat  Command 

• 

Air  National  Guard 

• 

Air  Force  Program  Executive  Office 

• 

Commander  of  ASC  and  OC-ALC 

• 

Other  Product  and  Logistics  Centers 

• 

Air  Force  Materiel  Command 

• 

HQ  USAF 

• 

Secretary  of  the  Air  Force  for  Acquisition 

• 

Office  of  the  Secretary  of  Defense 
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TABLE  1  (CONTINUED) 


B-2 

• 

Air  Combat  Command 

• 

509th  Bomb  Wing,  Whiteman  AFB,  MO 

• 

Air  Force  Materiel  Command 

• 

Air  Force  Program  Executive  Office 

• 

Secretary  of  the  Air  Force  for  Acquisition 

F-15 

• 

Air  Combat  Command 

• 

Air  Force  Materiel  Command 

• 

Air  Force  Program  Executive  Office 

• 

Other  SPOs 

• 

Air  Logistics  Centers 

• 

Test  Organizations 

• 

Persons  within  the  F-15  SPO 

F-16 

• 

Air  Combat  Command 

• 

Air  National  Guard 

• 

Air  Force  Reserves 

• 

Foreign  Military  Sales  (FMS)  Countries 

• 

European  Participating  Government  (EPG)  Countries 

• 

Air  Force  Materiel  Command 

• 

Air  Force  Program  Executive  Office 

• 

Secretary  of  the  Air  Force  for  Acquisition 

• 

Air  Logistics  Centers 

• 

Product  Centers 

• 

Congress 

F-22 

• 

Air  Combat  Command 

• 

Sacramento  Air  Logistics  Center 

• 

San  Antonio  Air  Logistics  Center 

• 

Air  Education  and  Training  Command  (AETC) 

• 

Air  Force  Program  Executive  Office 

• 

Secretary  of  the  Air  Force  for  Acquisition 

• 

Office  of  the  Secretary  of  Defense  (OSD) 

• 

Aeronautical  Systems  Center 

• 

Air  Force  Materiel  Command 

• 

Defense  Contractors 

• 

Congress 

C-17 

• 

Air  Mobility  Command 

• 

U.S.  Army 
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Research  Question  5.  What  characteristics  of  IPT  performance  do  IPT 
customers  perceive  as  most  important?  The  first  part  of  this  section  will  address  the 
customer  satisfaction  information  found  in  the  self-assessments  of  each  SPO.  In  addition, 
this  discussion  will  include  a  breakdown  of  the  actual  tools  (surveys)  used  by  each  SPO  to 
gather  customer  satisfaction  data. 

The  second  part  of  the  section  will  address  the  customer  satisfaction  information 
found  in  the  interviews  conducted  by  Brigadier  General  Kenne.  Specifically,  the  areas  that 
ASC  customers  have  identified  as  deficient  will  be  explored. 

B-l  Self-Assessment.  The  primary  tool  used  by  the  B-l  SPO  to  measure 
customer  satisfaction  is  the  Customer  Satisfaction  Assessment  Survey.  The  survey  is  used 
to  collect  data  relating  to  four  areas:  (1)  program  execution,  (2)  financial  management,  (3) 
product  acceptability,  and  (4)  working  relationships.  The  feedback  received  from  the 
surveys  is  used  to  focus  SPO  management  attention  in  areas  where  improvements  need  to 
be  made.  Table  2  depicts  the  actual  aspects  of  customer  satisfaction  focus  on  the  B-l 
Customer  Satisfaction  Assessment. 


TABLE  2 

B-l  CUSTOMER  SATISFACTION  DATA 


How  is  the  SPO  Doing? 

How  is  the  Overall  Program? 

• 

Overall  SPO  Performance 

• 

Overall  Contractor  Performance 

• 

Meeting  Management  Commitments 

• 

Technical  Performance 

• 

Communications 

• 

Schedule  Performance 

• 

Responsiveness 

• 

Cost  Performance 

• 

“Quality”  of  People  Working  Program 

• 

Overall  Product  Quality 

• 

“Quantity”  of  People  Working 

• 

Meeting  Commitments  Other  Than 

Program 

Product  Delivery 

• 

Keeping  YOU  Involved 

• 

Communication 
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In  addition  to  the  areas  listed  in  Table  2,  there  is  a  space  provided  to  note  any 
written  comments  or  suggestions  for  improving  customer  satisfaction. 

B-2  Self-Assessment.  The  B-2  SPO  does  not  have  a  customer  satisfaction 
instrument,  such  as  a  survey,  that  is  given  to  its  customers  on  a  periodic  basis.  However, 
the  B-2  SPO  has  developed  a  Capability  Assessment  Reference  Document  (CARD) 
database  that  enables  the  SPO  to  better  understand  and  work  the  resolution  of  problems 
that  limit  ACC’s  required  B-2  capabilities.  The  CARD  process  focuses  on  individual 
weapon  system  requirements  that  the  user  expects  the  SPO/contractor  team  to  meet. 

From  this  database,  the  B-2  SPO  focuses  its  customer  satisfaction  efforts  in  four  main 
areas:  (1)  closure,  or  removal,  of  high  priority  items  from  the  CARD,  (2)  consistent  and 
timely  responses  to  customer  inquiries,  (3)  the  taking  of  immediate  action  on  crisis  or 
safety-of-flight  problems,  and  (4)  the  elimination  or  mitigation  of  impacts  to  the 
production  line. 

F-15  Self-Assessment.  The  F-15  SPO  uses  several  methods  to  determine 
customer  satisfaction.  Methods  are  separated  according  to  customer  type  (external  and 
internal  customers  focus)  and  quantitative/qualitative  nature.  Methods  for  determining 
external  customer  satisfaction  include:  (1)  Customer  Satisfaction  Assessments,  (2) 
Customer  Site  Visits,  (3)  Quantitative  Metrics,  and  (4)  Management  Reviews.  Table  3 
lists  the  specific  areas  contained  within  each  of  the  external  customer  satisfaction  methods. 

In  addition  to  the  areas  listed  in  Table  3,  there  is  a  space  provided  to  note  any 
written  comments  or  suggestions  for  improving  customer  satisfaction. 
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Methods  for  determining  internal  customer  satisfaction  include:  (1)  ASC  Cultural 
Survey,  (2)  Continuous  Improvement  Team  (CIT)  survey,  and  (3)  IWSM  survey.  Table 
4  lists  the  specific  areas  contained  within  each  of  the  external  customer  satisfaction 
methods. 

In  addition  to  the  areas  listed  in  Table  4,  there  is  a  space  provided  to  note  any 
written  comments  or  suggestions  for  improving  customer  satisfaction. 


TABLE  3 

F-15  CUSTOMER  SATISFACTION  DATA  (EXTERNAL) 


Method 

Specific  Area  Addressed 

Customer  Satisfaction  Assessment 

• 

Depot  maintenance 

• 

User  involvement 

• 

Supply  support 

• 

Aircraft  support 

• 

Assessment  of  the  Program  Office 

• 

Determination  of  Program  cost 

• 

System  management  and  technical 
support 

Customer  Site  Visits 

• 

Interface  for  specific  products  and 
services 

Management  Reviews 

• 

Specific  cost,  schedule,  and 
performance  data  for  individual 
programs 

Quantitative  Metrics 

• 

Focus  on  effective  management  of 
systems 
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TABLE 4 


F-15  CUSTOMER  SATISFACTION  DATA  (INTERNAL) 


Method 

Specific  Area  Addressed 

ASC  Cultural  Survey 

• 

Examines  senior  and  mid-level  leadership  performance  in 
human  resource  management 

• 

Support  to  individuals  within  the  organization 

• 

Implementation  of  QAF  improvements 

CIT  Survey 

• 

Processes 

• 

Facilities 

• 

Management 

• 

Communication 

IWSM  Survey 

• 

TQM  implementation 

• 

Communication 

• 

Empowerment 

• 

Management 

• 

Customer  feedback 

• 

Quality  teams 

F-16  Self-Assessment.  The  F-16  SPO,  much  like  the  B-l  SPO,  uses  a 


Customer  Satisfaction  Assessment  Survey  as  a  means  to  determine  customer  satisfaction. 
The  survey  focuses  on  the  collection  of  data  relating  to  five  areas:  (1)  program  execution, 
(2)  financial  management,  (3)  product  acceptability,  (4)  working  relationships,  and  (5) 
other  topics.  The  specific  areas  of  interest  within  each  of  these  areas  are  depicted  in  Table 
5. 

In  addition  to  the  areas  listed  in  Table  5,  there  is  a  space  provided  to  note  any 
written  comments  or  suggestions  for  improving  customer  satisfaction. 


29 


TABLE  5 


F-16  CUSTOMER  SATISFACTION  DATA 


Program  Execution 

• 

Provides  you  with  viable  alternatives 

• 

Involves  you  in  technical  decisions 

• 

Involves  you  is  scheduling  decisions 

• 

Meets  its  scheduling  commitments 

• 

Involves  you  in  structuring  the  program  baseline 

• 

R&PC  provides  you  with  an  opportunity  to  identify 
new  requirement  and  air  frame  mission  deficiencies 

Program  Financial  Management 

• 

Involves  you  in  program  financial  exercises 

• 

Supports  your  budget  exercises 

• 

Provides  you  visibility  into  funding  status  and  needs 

• 

Manages  program  funds 

Product  Acceptability 

• 

Performance 

• 

Hardware  quality 

• 

Software  quality 

• 

Supportability 

• 

User  friendliness 

• 

Affordability 

Working  Relationships 

• 

Fosters  a  team  approach 

• 

Responds  to  your  communications 

• 

Provides  you  with  information 

• 

Emits  a  single  face  to  the  user  philosophy 

Other  Topics 

• 

Did  you  receive  the  product/service  you  asked  for? 

• 

Did  you  receive  it  on  time? 

• 

What  is  your  #1  concern  with  us? 

F-22  Self-Assessment.  The  F-22  SPO  sends  quarterly  surveys  to  its 
primary  customers.  The  purpose  of  the  survey  is  to  document  the  customer’s  satisfaction 
with  F-22  products  and  working  relationships.  The  survey  is  divided  into  three  sections. 
The  first  section  asks  two  primary  questions  that  the  SPO  would  like  answered:  (1)  How 
satisfied  are  you  with  F-22  products?  (2)  How  well  is  the  Program  Office  working  with 
you?  The  second  section  is  an  optional  section  for  the  customer  to  provide  comments  on 
specific  areas  of  the  program.  These  areas  are  listed  in  Table  6. 
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TABLE  6 


F-22  CUSTOMER  SATISFACTION  DATA 


f  F-22  Program  Assessment  Indicators  -  Comments/Concems 

•  Performance  Characteristics 

•  Test  and  Evaluation 

4 

•  Logistics  Requirements  and  Sustainment  Objectives 

•  Cost 

•  Funding 

•  Schedule 

•  Contracts 

•  Production 

•  Management  Structure 

•  Reports/Briefings _ 


The  third  section  of  the  survey  lists  the  SPO’s  top  five  program  priorities  and  asks 
the  customer  to  provide  its  top  program  priorities. 

C-17  Self- Assessment.  The  C-17  SPO  does  not  rely  on  comment  cards  or 
questionnaires  to  determine  customer  satisfaction.  It  is  their  belief  that  they  have  failed 
when  the  customer  finds  that  it  needs  to  formally  write  regarding  lack  of  attention  to  the 
issues.  Rather,  they  rely  on  daily  face-to-face  interchanges  at  all  levels  of  the  program 
office  to  search  out  better  ways  of  understanding  and  meeting  the  needs  of  the  customer. 
The  key  to  this  interaction  is  the  complete  exchange  of  information  to  ensure  that  the 
issues  and  needs  of  the  customer  are  fully  understood.  The  C-17  SPO  defines  customer 
satisfaction  as  meeting  product  and  delivery  requirements. 

Kenne  Interviews.  Brigadier  General  Kenne’s  interviews  with  the  Air 
Mobility  Command,  Air  Combat  Command,  Air  Education  and  Training  Command,  Air 
Force  Special  Operations  Command,  and  Headquarters  Air  Force  Reserves  revealed  three 
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areas  as  being  of  particular  importance.  Those  three  areas  are:  (1)  communications,  (2) 
responsiveness,  and  (3)  strategic  planning.  The  customer  concerns  in  the  area  of 
communications  centered  on  earlier  user  involvement,  more  information  on  tradeoff 
possibilities,  and  more  effective  feedback  on  contractual  and  financial  matters.  In  regard 
to  responsiveness,  the  customers  emphasized  the  need  for  better  support  in  the  case  of 
short-turnaround  requirements  and  “what-if  ’  exercises.  Finally,  the  customer  concerns  in 
the  area  of  strategic  planning  revolved  around  more  effective  up-front  planning  regarding 
the  feasibility  of  a  solution,  more  aggressive  and  effective  risk  management,  and  a  more 
proactive  approach  in  anticipating  issues  before  they  become  problems. 

In  addition,  Brigadier  General  Kenne  also  had  the  MAJCOMs  prioritize  the  more 
traditional  indicators  of  cost,  schedule,  and  performance.  Interestingly,  in  a  time  of  severe 
budget  constraints,  performance  is  still  the  number  one  priority  of  the  MAJCOMs,  with 
cost  and  schedule  rated  relatively  the  same,  but  below  performance. 

The  MAJCOMs  were  also  asked  to  identify  any  other  items  they  considered  to  be 
better  indicators  than  cost,  schedule,  and  performance.  The  three  additional  indicators 
identified  were:  (1)  life  cycle  cost  reduction,  (2)  degree  of  commonality,  and  (3)  mobility 
footprint.  These  three  indicators  represent  a  very  logistics-oriented  and  “big  picture” 
approach  to  the  acquisition  of  future  weapon  systems.  While  life  cycle  cost  reduction  is 
always  a  consideration  in  the  acquisition  of  a  new  weapon  system,  the  MAJCOMs 
indicated  a  strong  desire  to  place  greater  emphasis,  up  front,  on  this  issue.  Degree  of 
commonality  refers  to  the  use  of  common  parts/systems  between  major  weapon  systems. 
Concentrated  efforts  to  develop  systems  that  could  be  used  interchangeably  might  cost 
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more  on  the  front  side,  but  will  decrease  the  overall  life  cycle  costs  and  shorten  logistical 
support  lines  required  to  maintain  the  particular  weapon  systems.  And  finally,  mobility 
footprint  refers  to  the  time  and  support  needed  to  get  a  weapon  system  to  a  fully 
deployable  stage;  it  focuses  on  the  all  aspects  of  logistical  support. 

Research  Question  6.  Of  those  characteristics  identified  by  the  IPTs  and 
their  customers  as  most  important,  which  are  the  most  appropriate  as  a  foundation 
for  the  development  of  metrics  for  measuring  the  effectiveness  of  IPTs?  As  discussed 
in  Chapter  in,  our  approach  to  analyzing  the  data  gathered  from  the  self-assessments  and 
the  Kenne  interviews  will  entail  a  merging,  or  comparison,  of  the  inputs  from  both 
sources.  By  merging  the  inputs  from  both  sources,  a  common  ground  between  the 
existing  criteria  and  the  actual  customer  needs  can  be  established.  In  turn,  this 
commonality  will  isolate  those  characteristics  that  are  identified  as  being  most  critical  to 
customer  satisfaction  from  the  standpoint  of  both  the  IPTs  and  their  customers.  The 
characteristics  which  can  be  isolated  will  then  be  identified  as  the  key  characteristics 
against  which  metrics  should  be  developed  for  the  purpose  of  measuring  the  effectiveness 
of  IPTs. 

A  comparison  of  the  data  gathered  from  the  two  sources  reveals  some  interesting 
points:  (1)  the  traditional  indicators  of  cost,  schedule,  and  performance  are  still  of 
considerable  importance  to  the  customers;  (2)  there  are  other  indicators,  such  as 
communication  and  responsiveness,  that  are  common  throughout  the  self-assessments  and 
the  Kenne  interviews;  (3)  there  is  a  movement  toward  logistics-oriented  indicators  in  both 
the  F-15  and  F-22  self-assessments,  as  well  as  the  Kenne  interviews;  and  (4)  there  appears 
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to  be  a  desire  for  more  emphasis  on  strategic  planning  on  the  part  of  the  customers  (Kenne 

interviews)  that  does  not  seem  to  be  reflected  in  the  SPO  self-assessments. 

The  emphasis  on  cost,  schedule,  and  performance  is  evident  throughout  the  self- 

assessments  and  the  Kenne  interviews.  There  are  currently  program  office  metrics  in  place 

based  on  these  traditional  indicators,  and  any  attempt  to  develop  metrics  for  the  purpose 

of  measuring  IPT  effectiveness  cannot  discard  cost,  schedule,  and  performance  as  relevant 

bases  for  a  performance  measurement  system.  There  are,  however,  other  indicators  which 

are  more  relevant  to  the  team  approach  fostered  by  the  IWSM  philosophy.  Two  of  these 

which  show  up  often  in  the  data  gathered  are  communication  and  responsiveness.  These 

two  indicators  are  very  consistent  with  the  tenets  of  IWSM  and  the  concepts  upon  which 

the  IPT  approach  is  founded.  The  IPT  approach  is  founded  on  communication  between 

the  parties  and  responsiveness  of  the  IPTs  to  the  concerns  and  needs  of  the  customer. 

“It  is  important  to  get  customer  involvement  as  early  as  possible.  Customers  may 
have  input  or  issues  that  will  impact  the  way  the  single  manager  will  conduct 
business.  They  are  as  much  a  part  of  the  overall  team  as  program  office  personnel, 
and  their  early  participation  is  essential.”  (AFMC,  1993:  22) 

“A  single  manager  is  empowered  with  the  maximum  authority  over  the  widest 
range  of  program  decisions  and  resources  to  satisfy  customer’s  requirements.  This 
gives  the  user  a  single  individual  to  address  any  issues  that  need  to  be  worked  to 
ensure  our  war  fighters  are  supported.”  (AFMC,  1993:  12) 

Continuing  to  explore  communication  and  responsiveness,  only  one  organization 

appears  to  dedicate  considerable  effort  to  determining  the  satisfaction  of  IPT  members  as 

internal  customers.  The  F-15  SPO  has  employed  a  lengthy,  computer-based  survey  that  is 

used  to  determine/solicit  the  views  and  feelings  that  exist  in  the  SPO  environment. 

Although  many  of  the  organizations,  including  the  F-15,  use  the  ASC  Cultural  Survey  to 
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evaluate  the  working  climate  of  the  whole  organization,  an  additional  measurement  system 
that  solicits  the  individual  thoughts  of  IPT  members  may  provide  valuable  information  that 
will  enhance  a  SPO’s  ability  to  effectively  build  solid  teams  (IPTs)  in  the  future.  “One  of 
the  most  critical  areas  of  a  program’s  development  and  execution  is  forming  a  solid, 
cohesive  team  early  in  the  integration  process”  (AFMC,  1993:  29). 

The  movement  toward  logistics-oriented  indicators  that  is  evident  in  the  F-15  and 
F-22  self-assessments,  as  well  as  the  Kenne  interviews,  appears  to  represent  a  realization 
on  the  part  of  the  IPTs  and  their  customers  of  the  cost  impact  associated  with  the  logistics 
support  of  a  system.  Under  the  IWSM  philosophy,  the  single  manager  is  the  single 
individual  who  ties  together  two  formerly  separate  functions:  systems  acquisition  and 
sustainment. 

“The  single  manager,  because  of  the  nature  of  the  life  cycle  responsibilities  that  are 
inherent  in  the  IWSM  concept,  will  be  key  to  establishing  the  partnership 
between  acquisition  and  sustainment.”  (AFMC,  1993:  14) 

Given  the  budget  constraints  that  currently  exist  and  the  significant  portion  of  the 

cost  of  a  weapon  system  that  can  be  attributed  to  logistics  support,  the  movement  toward 

logistics-oriented  indicators  may  be  one  that  will  continue,  and  probably  expand,  in  the 

future. 

The  desire  of  the  customers  to  increase  the  emphasis  on  strategic  planning  is  not 
surprising  given  the  strategic  planning  movement  currently  underway  within  the  Federal 
Government.  The  Government  Performance  and  Results  Act  mandates  the  use  of  a 
strategic  planning  approach  within  all  Federal  organizations.  The  strategic  plans  will 
cover  a  minimum  of  five  years  and  be  updated  at  least  every  three  years.  Each 
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organization’s  mission  is  the  cornerstone  of  its  strategic  plan.  “Once  its  general  goals  and 
objectives  are  developed,  it  is  necessary  to  set  specific,  objective,  measurable  goals” 

(United  States  Senate,  1993:  15).  These  goals  must  be  tied  to  the  organizations  day-to-  > 

day  activities  and  must  drive  its  performance.  With  the  emergence  of  such  guidance,  and 

* 

the  benefits  to  be  realized  through  such  an  approach,  strategic  planning  will  become  of 
even  greater  importance  to  the  customers  of  IPTs. 

Based  on  the  above  discussion,  seven  characteristics  were  identified  as  being  the 
most  appropriate  as  a  foundation  for  the  development  of  metrics  for  measuring  the 
effectiveness  of  IPTs.  Those  seven  characteristics  must  be  the  foundation  for  any  IPT 
performance  measurement  system.  The  seven  characteristics,  and  the  rationale  for  their 
selection,  are  discussed  in  detail  in  Chapter  V. 
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V.  Summary,  Conclusions,  and  Recommendations 


Summary 

i 

Chapter  I  of  this  research  effort  introduced  the  problems  encountered  by  the  Air 
*  Force  in  merging  AFSC  and  AFLC.  These  problems  drove  a  new  management  philosophy 

known  as  Integrated  Weapon  System  Management  (IWSM).  IWSM  “empowered  a 
single  manager  with  authority  over  the  widest  range  of  decisions  and  resources  to  satisfy 
customer  requirements  throughout  the  life  cycle  of  the  product”.  Out  of  the  IWSM 
philosophy,  came  the  concept  of  Integrated  Product  Teams  (IPTs).  IPTs  are  “cross¬ 
functional  teams  that  are  formed  for  the  specific  purpose  of  delivering  a  product  or 
managing  a  process  for  the  customer”.  Based  on  the  direction  of  General  Fain,  IPTs 
became  the  primary  working  unit  within  ASC.  The  switch  to  an  IPT  approach  within 
ASC  made  it  imperative  that  a  performance  measurement  system  be  established  in  order  to 
accurately  assess  the  quality  of  IPT  performance.  There  is  currently  no  such  measurement 
system  within  ASC. 

The  focus  of  this  research  is  on  the  identification  of  characteristics  which  can 

provide  the  foundation  for  the  establishment  of  a  performance  measurement  system  that 

can  accurately  assess  the  effectiveness  of  IPTs.  Since  customer  satisfaction  is  the  key  to 

the  management  philosophy  implemented  by  IWSM,  it  must  also  be  the  central  focus  in 

the  development  of  the  performance  measurement  system.  Data  regarding  those 
r 

characteristics  considered  most  important  to  the  IPTs  and  their  customers  was  gathered 

4 

primarily  from  two  sources:  (1)  Chapter  7  of  the  Quality  Air  Force  Self-Assessments  for 
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the  B-l,  B-2,  F-15,  F-16,  F-22,  and  C-17  SPOs  and  (2)  personal  interviews  conducted  by 

Brigadier  General  Kenne  with  five  major  customers  of  ASC.  The  data  gathered  was  then 

subjected  to  an  analysis  to  determine  those  characteristics  most  appropriate  as  a  > 

foundation  for  the  development  of  a  performance  measurement  for  assessing  the 

effectiveness  of  IPTs.  By  merging  the  inputs  from  both  data  sources,  a  common  ground 

between  the  existing  criteria  and  the  actual  customer  needs  was  established.  In  turn,  this 

commonality  enabled  the  researchers  to  isolate  those  characteristics  that  are  identified  as 

being  most  critical  to  customer  satisfaction  from  the  standpoint  of  both  the  IPTs  and  the 

customers.  The  seven  characteristics  which  were  isolated  in  this  manner  were  then 

identified  as  the  key  characteristics  against  which  metrics  should  be  developed  for  the 

purpose  of  measuring  the  effectiveness  of  EPTs. 

Conclusions 

The  seven  characteristics  identified  as  being  the  most  appropriate  as  a  foundation 
for  the  development  of  metrics  for  measuring  the  effectiveness  of  IPTs  were:  (1)  cost,  (2) 
schedule,  (3)  performance,  (4)  communication,  (5)  responsiveness,  (6)  logistics  support, 
and  (7)  strategic  planning.  These  characteristics  have  been  identified  through  the  analysis 
of  data  gathered  from  the  six  SPO  self-assessments  and  interviews  conducted  by  Brigadier 
General  Kenne  with  five  major  customers  of  ASC.  The  characteristics  identified  were 
chosen  primarily  because:  (1)  they  appeared  consistently  in  both  the  SPO  self-assessments 
and  the  Kenne  interviews  and  (2)  they  were  consistent  with  both  the  IWSM  philosophy 

* 

and  current  trends  in  Federal  Government  and  Air  Force  policy.  The  ultimate  selection  of 
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the  seven  characteristics  certainly  required  some  degree  of  subjectivity  on  the  part  of  the 
researchers.  However,  the  patterns  prevalent  in  the  data  showed  clear  trends  as  to  the 
preferences  of  the  customers  of  ASC.  Each  of  the  seven  characteristics  will  be  discussed 
in  greater  detail  in  the  following  paragraphs. 

As  discussed  in  Chapter  IV,  the  emphasis  on  cost,  schedule,  and  performance 
was  evident  throughout  both  the  SPO  self-assessments  and  the  Kenne  interviews.  These 
three  characteristics  were  found  in  all  six  of  the  SPO  self-assessments,  as  well  as  being  a 
key  topic  of  discussion  in  the  Kenne  interviews.  While  the  initial  intent  of  the  researchers 
was  to  evolve  away  from  the  more  traditional  characteristics  such  as  cost,  schedule,  and 
performance,  the  strong  presence  of  these  characteristics  in  the  data  gathered  made  it 
imperative  that  they  be  included  in  the  relevant  database  as  a  potential  foundation  for  the 
development  of  IPT  metrics.  Currently,  there  are  program  office  metrics  in  place  based  on 
these  traditional  indicators,  however  the  current  metrics  must  be  revised  to  be  more 
compatible  with  the  IPT  philosophy. 

Communication  and  responsiveness  were  two  other  characteristics  that  showed 
up  consistently  in  the  data  gathered.  These  characteristics  were  found  in  the  B-l,  B-2,  F- 
15,  and  C-17  self-assessments,  as  well  as  the  Kenne  interviews.  The  consistent  presence 
of  these  two  characteristics  in  the  data,  as  well  as  the  relationship  of  the  characteristics  to 
the  overall  intent  of  the  IWSM  approach,  made  it  imperative  that  both  communication  and 
responsiveness  be  included  in  the  relevant  database  as  a  potential  foundation  for  the 
development  of  IPT  metrics.  The  IWSM  approach  is  founded  on  communication  between 
the  parties  and  responsiveness  of  the  IPTs  to  the  concerns  and  needs  of  the  customer.  The 
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development  of  a  performance  measurement  system  for  IPTs  without  including  these  two 
characteristics  would  be  inconsistent  with  the  elements  of  IWSM  and  counterproductive 
to  continuous  improvement  efforts. 

It  was  also  evident  from  the  data  gathered  that  logistics  support  was  a 
characteristic  that  was  of  considerable  concern  to  the  customers  of  ASC.  This 
characteristic  was  found  in  the  F-15  and  F-22  self-assessments,  as  well  as  the  Kenne 
interviews.  Again,  this  characteristic  is  consistent  with  the  overall  IWSM  philosophy  and 
the  goal  of  that  philosophy  to  present  a  single  manager  who  is  responsible  for  both 
systems  acquisition  and  sustainment.  This  relationship  to  the  IWSM  philosophy,  and  the 
significant  presence  of  the  characteristic  in  the  data,  made  it  imperative  that  logistics 
support  be  included  in  the  relevant  database. 

The  final  characteristic  identified  as  being  most  appropriate  as  a  foundation  for  the 
development  of  metrics  for  measuring  the  effectiveness  of  IPTs  was  strategic  planning. 
The  only  presence  of  this  characteristic  in  the  data  gathered  was  in  the  Kenne  interviews. 
However,  the  strong  presence  there  and  the  significant  emphasis  being  placed  on  strategic 
planning  within  the  Federal  Government  made  it  imperative  that  this  characteristic  be 
included  in  the  relevant  database.  As  discussed  in  Chapter  IV,  the  Government 
Performance  and  Results  Act  mandates  the  use  of  a  strategic  planning  approach  within  all 
Federal  organizations.  Such  a  mandate  ensures  that  strategic  planning  will  be  become  an 
even  greater  concern  to  the  customers  of  IPTs  and  should  be  of  considerable  importance 
in  the  development  of  any  performance  measurement  system  for  IPTs. 
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The  identification  of  these  characteristics  is  only  the  first  step  in  the  overall  process 
of  developing  an  IPT  performance  measurement  system.  The  characteristics  represent  an 
f  important  initial  building  block  in  the  development  of  metrics  to  measure  IPT 

effectiveness.  The  potential  for  further  research  into  the  characteristics  identified  and  how 
they  can  be  evolved  into  specific  metrics  for  measuring  IPT  effectiveness  is  certainly 
present. 

The  identification  of  these  characteristics  will  enable  future  researchers  to  focus 
their  efforts  on  developing  metrics  for  assessing  IPT  performance,  rather  than  attempting 
to  isolate  those  characteristics  against  which  the  metrics  should  be  developed.  Such  focus 
should  expedite  the  development  of  an  effective  IPT  performance  measurement  system. 

Recommendations 

As  discussed  above,  there  is  considerable  potential  for  further  research  into  the 
characteristics  identified  by  this  research  and  how  they  can  be  evolved  into  specific  metrics 
for  measuring  IPT  effectiveness.  In  order  to  proceed  beyond  this  initial  step  and  develop 
actual  metrics  for  assessing  IPT  performance,  future  researchers  must  focus  their  attention 
on  the  characteristics  identified  herein  and  the  criteria  for  effective  metrics  which  have 
previously  been  explored  in  Chapter  H  Additional  research  must  be  conducted  within  the 
IPT  environment  to  ensure  that  the  metrics  developed  are  meaningful,  repeatable,  timely 
and,  most  importantly,  capable  of  driving  the  appropriate  action.  The  seven  areas 
identified  in  this  research  must  be  broken  down  even  further  to  discover  the  actual 

4 

relationships  that  occur  between  the  IPTs  and  their  customers.  For  example, 
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communication  is  a  very  broad  subject.  When  a  customer  wishes  to  improve 
communication  he/she  is  probably  not  in  need  of  a  more  sophisticated  telephone.  What 
type  of  communication  needs  improvement?  Does  the  customer  desire  more  input  into 
decision-making?  Does  he/she  just  want  to  know  the  contractor’s  schedule  status? 
Breaking  communication  down  into  categories  such  as  (1)  information  (providing  daily, 
monthly,  etc.,  reports),  (2)  decision-making  (requests  for  guidance/advice),  and  (3) 
approvals  (coordinating  mandatory  actions  with  the  appropriate  approving  authority) 
would  be  the  next  logical  step  into  developing  effectiveness  measures  for  IPTs. 
Regardless  of  what  the  actual  subcategories  of  communication  are,  future  research  in  this 
area  will  dictate  the  need  to  work  very  closely  with  functioning  IPTs.  Future  researchers 
will  almost  need  to  become  members  of  an  IPT  to  understand  the  day-to-day  interfaces 
that  occur  between  IPTs  and  their  customers.  By  being  so  closely  related  to  a  team,  the 
future  researcher  can  then  discriminate,  through  experience,  between  critical  customer 
interactions  and  daily  administrative  actions. 
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